Learning to execute algorithms is a fundamental problem that has been widely studied. Prior work (Veličković et al., 2019) has shown that to enable systematic generalisation on graph algorithms it is critical to have access to the intermediate steps of the program/algorithm. In many reasoning tasks, where algorithmic-style reasoning is important, we only have access to the input and output examples. Thus, inspired by the success of pre-training on similar tasks or data in Natural Language Processing (NLP) and Computer vision, we set out to study how we can transfer algorithmic reasoning knowledge. Specifically, we investigate how we can use algorithms for which we have access to the execution trace to learn to solve similar tasks for which we do not. We investigate two major classes of graph algorithms, parallel algorithms such as breadth-first search and Bellman-Ford and sequential greedy algorithms such as Prims and Dijkstra. Due to the fundamental differences between algorithmic reasoning knowledge and feature extractors such as used in Computer vision or NLP, we hypothesis that standard transfer techniques will not be sufficient to achieve systematic generalisation. To investigate this empirically we create a dataset including 9 algorithms and 3 different graph types. We validate this empirically and show how instead multi-task learning can be used to achieve the transfer of algorithmic reasoning knowledge.

Large Language Models (LLMs) for Graph Reasoning have been extensively studied over the past two years, involving enabling LLMs to understand graph structures and reason on graphs to solve various graph problems, with graph algorithm problems being the most prevalent. Recent studies underscore the potential of LLMs in handling graph reasoning tasks, but their performance is underwhelming. In this work, we point out issues with existing methods and benchmarks, and rethink the direction that LLMs for graph reasoning should strive toward. We find that base models, e.g., GPT-4o-mini, are largely underestimated due to improper reasoning focus. Base models with reasoning focus redirected from replicating graph algorithms to designing them can easily solve most graph reasoning tasks in existing benchmarks. To truly evaluate the graph reasoning capabilities of LLMs, we construct a more challenging GraphAlgorithm benchmark, comprising 239 different graph problems and 3,041 test instances collected from 4 competition platforms. Finally, we introduce a simple and strong baseline Simple-Reasoning-Then-Coding (Simple-RTC)-which guides LLMs to design graph algorithms first and then code to address graph reasoning tasks. Simple-RTC achieves near-perfect accuracy on existing benchmarks and significantly outperforms GPT-4o-mini and all prior methods on the GraphAlgorithm benchmark. This strong baseline encourages further advancements in LLMs for Graph Reasoning in the future.

Hyperparameters V alue Number of encoder (decoder) layers 6 Number of layers in the feed forward network 2 Number of hidden units in the feed forward network 128 Mask filter size 3 Mask number of filters 16 Ratio of residual connection 1.5 Dropout rate 0.1 Optimizer Adam Warm-up steps 4000 Learning rate p d min ( p t, t 4000 Unless otherwise specified, the task performed in this section is selection sort (Section 4). Figure 6 shows the sorting performance of the transformers w/o mask supervision. Figure 7 shows sorting performances with different encoding schemes. In Figure 9, we show the strong generalization performance of the different architectures. While some changes are able to improve performance in this regime, the performance ultimately drops steeply as the length of the test sequence increases. The symbol e represents the end token.

Learning to execute algorithms is a fundamental problem that has been widely studied.

Graph-structured data are pervasive across domains including social networks, biological networks, and knowledge graphs. Due to their non-Euclidean nature, such data pose significant challenges to conventional machine learning methods. This study investigates graph data structures, classical graph algorithms, and Graph Neural Networks (GNNs), providing comprehensive theoretical analysis and comparative evaluation. Through comparative experiments, we quantitatively assess performance differences between traditional algorithms and GNNs in node classification and clustering tasks. Results show GNNs achieve substantial accuracy improvements of 43% to 70% over traditional methods. We further explore integration strategies between classical algorithms and GNN architectures, providing theoretical guidance for advancing graph representation learning research.